feature: Add Weighted Least Squares (WLS) IVIM fitting algorithm — DT_IIITN (Feature #110)#148
feature: Add Weighted Least Squares (WLS) IVIM fitting algorithm — DT_IIITN (Feature #110)#148Devguru-codes wants to merge 3 commits intoOSIPI:mainfrom
Conversation
Implement Weighted Least Squares (WLS) segmented IVIM fitting following Veraart et al. (2013) NeuroImage 81:335-346. Algorithm: - Step 1: Fit D from high b-values via WLS on log-signal (w=S^2) - Step 2: Fit D* from residuals at low b-values via WLS New files: - src/original/DT_IIITN/wls_ivim_fitting.py (raw algorithm, numpy only) - src/standardized/DT_IIITN_WLS.py (OsipiBase standardized wrapper) Modified files: - tests/IVIMmodels/unit_tests/algorithms.json (register for automated tests) Follows repository contribution structure: - Fit code in src/original/Initials_Institution/ - Standardized wrapper in src/standardized/ - Registered in algorithms.json (no custom tests needed) Test results: 1184 passed, 167 skipped, 22 xfailed, 6 xpassed. 27 test_volume errors are pre-existing (FileNotFoundError on Windows).
src/standardized/DT_IIITN_WLS.py
Outdated
| required_bvalues = 4 | ||
| required_thresholds = [0, 0] | ||
| required_bounds = False | ||
| required_bounds_optional = True |
There was a problem hiding this comment.
@IvanARashid , I'm a bit confused what the difference is between
requiered bounds
requiered bounds optional
supported bounds
Does it make sense for the second to be true and the others false?
There was a problem hiding this comment.
I followed the src\standardized\ETP_SRI_LinearFitting.py file pattern which also has required_bounds_optional = True with supported_bounds = False. From what i understood, required_bounds_optional = True means means the wrapper's init file will accept bounds without raising an error, while supported_bounds = False means the underlying algorithm doesn't actually use them. If any suggestion come up, I will update this. Thank you.
There was a problem hiding this comment.
Yeah, not the wisest naming convention on my part. required_bounds can be False while required_bounds_optional can be true for algorithms that e.g. don't require bounds but can still take them as inputs. But I think both should be false for ETP_SRI_LinearFitting since it doesn't look like any bounds are being passed to the algorithm. It's something that has been overlooked in our previous passes.
So required_bounds_optional just means if bounds are optional and can still be passed.
There was a problem hiding this comment.
Keep in mind that LinearFitting was one of the very first algorithms to be committed to the repo and standardized, so there can be variables there that are deprecated :)
There was a problem hiding this comment.
Thank you @IvanARashid sir for helping here. My algorithm here doesnt use bounds and initial guesses, I will set required_bounds_optional = False and required_initial_guess_optional = False as well. Thank you.
There was a problem hiding this comment.
Keep in mind that LinearFitting was one of the very first algorithms to be committed to the repo and standardized, so there can be variables there that are deprecated :)
Do we need to update it or should we let it be. If you think I should update it, please tell. I will do it. Thank you.
|
@Devguru-codes You say it's easy to swap between WLS and RLM. Would it be too hard to allow either one to be used by way of an input? |
I have implemented it now by adding method parameter, the user can do method="RLM" as an input to seamlessly swap between the two. Thank you. |
|
I have implemented all the changes suggested by experts. Please check it @oliverchampion sir. Thank you. |
Hello @oliverchampion. The man who was assigned this issue didnt update his PR and had issues in it so I worked according to "how to commit" given in readme. I have also taken 1 decision explained in this PR (
statsmodels.robust.robust_linear_model.RLMvs numpy WSL) . If you suggest that we should use the former than latter - please let me know and I will update this PR.What this PR does
Closes #110. Implements a Weighted Least Squares (WLS) segmented IVIM fitting algorithm in accordance with the repository's contribution structure, following the approach recommended in Veraart et al. (2013).
This PR addresses the rejected PR #136, which implemented the same feature but placed code in the wrong location. This PR follows the correct structure:
src/original/Initials_Institution/src/standardized/algorithms.json(no custom test file needed)Algorithm
Segmented two-step WLS approach:
Step 1 — Estimate D (high b-values, b ≥ 200 s/mm²):
At high b-values the perfusion component decays to ~0, leaving:
Weighted linear regression on log-signal with weights w = S(b)² (Veraart correction for heteroscedasticity introduced by the log-transform). Intercept yields f.
Step 2 — Estimate D* (low b-values):
Subtract the diffusion component to isolate the perfusion signal:
Weighted linear regression with weights w = residual² gives D*.
Implementation: Pure
numpy(no extra dependencies). Uses the closed-form weighted normal equations:Files Changed
src/original/DT_IIITN/__init__.pysrc/original/DT_IIITN/wls_ivim_fitting.pysrc/standardized/DT_IIITN_WLS.pytests/IVIMmodels/unit_tests/algorithms.jsonDT_IIITN_WLSAPI Usage
Design Decision: numpy WLS vs statsmodels RLM
The issue description suggests using
statsmodels.robust.robust_linear_model.RLM. After evaluating both approaches:statsmodelsThe numpy WLS approach implements the same weighting strategy recommended by Veraart et al. (w = S²) but using the closed-form normal equations — which is exactly what the paper validates. The
statsmodels.RLMiterative approach adds robustness to outliers but at a ~15× runtime cost that is impractical for voxel-wise fitting on large datasets.If you (anyone who will see my pr) prefers the full RLM approach, it can be easily switched in
wls_ivim_fitting.pyby replacing the_weighted_linregcall withsm.RLM(...).fit().Reference
Veraart, J. et al. (2013). "Weighted linear least squares estimation of diffusion MRI parameters: strengths, limitations, and pitfalls." NeuroImage, 81, 335–346.
DOI: 10.1016/j.neuroimage.2013.05.028
Testing
Testing is handled automatically by the existing framework via
algorithms.jsonregistration.DT_IIITN_WLS)test_ivim_fit_savedtest_volume/test_parallelerrorsmain(WindowsFileNotFoundErrorin subprocess, unrelated to this PR — confirmed by running same tests onmainbefore this branch)Sample fit result on synthetic signal (
f=0.10, D=0.0010, D*=0.010):Checklist
src/original/Initials_Institution/+src/standardized/contribution structurealgorithms.jsonnumpyonly —statsmodelsevaluated but removed for performance; see Design Decision above)